Method and apparatus for evaluating predictive model

ABSTRACT

An approach for a computer to evaluate a predictive model includes identifying features of training samples in a set of training samples. The approach selects evaluation metrics from a set of evaluation metrics as available metrics using identified features and includes determining recommended metrics using the predictive model, the available metrics, and a predetermined set of user-preferred metrics. The approach applies the predictive model created using the set of training samples to a set of test samples to calculate values of the available metrics. The approach evaluates the predictive model by using the available metrics and the values of the available metrics to evaluate the predictive model by evaluating the predictive model using the recommended metrics and the values of the recommended metrics.

FIELD OF THE INVENTION

Embodiments of the present invention relate to the data mining field,and more specifically, to a method and apparatus for evaluating apredictive model.

BACKGROUND OF THE INVENTION

Data mining (DM), also termed knowledge discovery in database (KDD), isa hot issue of research in the current artificial intelligence anddatabase field. The data mining refers to a process of discovering fromhuge data of database implicit, previously unknown and potentiallyvaluable information. Generally, the data mining is the process ofautomatically searching in huge data for information having specialrelationships (belonging to association rule learning) and hiddentherein. The data mining is usually associated with computer science andaccomplishes the objects as described above by means of such asstatistics, online analysis processing, information retrieval, machinelearning, expert system (relying on past rules of thumb), and patternrecognition.

Various data mining platforms have been developed so far, by whichpredictive models can be created rapidly and applied to industries so asto help decision-makers to make correct decisions. Manifestation formsof predictive models may include rule sets, mathematical formulas,decision trees, etc, which may be used to generate prediction resultsaccording to a group of inputs or variables. After creating a predictivemodel, performance (precision) of the predictive model needs to beevaluated by using evaluation metrics, so as to ensure precision of thegenerated prediction results.

There exist various metrics for evaluating predictive models, such asreceiver operating area under curve (AUC), accuracy, F-score, recall,precision, etc. However, data flow platforms that have been developedcan only provide such an evaluation metric as accuracy. In some cases,for example with respect to unbalanced samples, the metric “accuracy” isnot sufficient to reflect performance of the created predictive model.Therefore, it is a research focus in the current data mining fieldregarding how to determine from multiple evaluation metrics one or moresuitable evaluation metrics for evaluating a predictive model.

SUMMARY

Embodiments of the present invention disclose a method, a computerprogram product, and a system for one or more computer processors toevaluate a predictive model. The method includes one or more computerprocessors identifying features of training samples in a set of trainingsamples. The method includes one or more computer processors selectingat least one evaluation metric from a set of evaluation metrics as oneor more available metrics based on the identified features whereselecting includes determining one or more recommended metrics based onthe predictive model, the one or more available metrics, and apredetermined first set of user-preferred metrics. Additionally, themethod includes one or more computer processors applying the predictivemodel created based on the set of training samples to a set of testsamples so as to calculate values of the one or more available metrics,where applying a predictive model created based on the set of trainingsamples to a set of test samples to calculate values of the one or moreavailable metrics includes determining values of the one or morerecommended metrics based on the values of the one or more availablemetrics. Furthermore, the method includes one or more computerprocessors evaluating the predictive model by using the one or moreavailable metrics and the values of the one or more available metrics,wherein evaluating the predictive model comprises evaluating thepredictive model by using the one or more recommended metrics and thevalues of the one or more recommended metrics.

BRIEF DESCRIPTION OF THE DRAWINGS

Through the more detailed description of some embodiments of the presentdisclosure in the accompanying drawings, the above and other objects,features and advantages of the present disclosure will become moreapparent, wherein the same reference generally refers to the samecomponents in the embodiments of the present disclosure.

FIG. 1 schematically shows a block diagram of an exemplary computersystem which is applicable to implement embodiments of the presentinvention;

FIG. 2 schematically shows a flowchart of a method for evaluating apredictive model according to an embodiment of the present invention;

FIG. 3 schematically shows a flowchart of a process for identifyingfeatures of training samples and selecting available metrics from a setof evaluation metrics based on the identified features according to anembodiment of the present invention;

FIG. 4 schematically shows a flowchart of a method for evaluating apredictive model according to another embodiment of the presentinvention;

FIG. 5 schematically shows a flowchart of a method for evaluating apredictive model according to a further embodiment of the presentinvention;

FIG. 6 schematically shows a block diagram of an apparatus forevaluating a predictive model according to an embodiment of the presentinvention;

FIG. 7 depicts Table 1 showing an exemplary set of samples in anembodiment of the present invention;

FIG. 8 depicts Table 2 showing an example of a set of evaluation metricsin an embodiment of the present invention;

FIG. 9 depicts Table 3 showing an example of calculating values ofavailable metrics according to an embodiment of the present invention;

FIG. 10 depicts Table 4 showing an example of a first set ofuser-preferred metrics in an embodiment of the present invention; and

FIG. 11 depicts Table 5 showing an example of a second set ofuser-preferred metrics in an embodiment of the present invention.

DETAILED DESCRIPTION OF EMBODIMENTS

Some preferable embodiments will be described in more detail withreference to the accompanying drawings, in which the preferableembodiments of the present disclosure have been illustrated. However,the present disclosure can be implemented in various manners, and thusshould not be construed to be limited to the embodiments disclosedherein. On the contrary, those embodiments are provided for the thoroughand complete understanding of the present disclosure, and completelyconveying the scope of the present disclosure to those skilled in theart.

As will be appreciated by one skilled in the art, aspects of the presentinvention may be embodied as a system, method or computer programproduct. Accordingly, aspects of the present invention may take the formof an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.) or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” or “system.”Furthermore, aspects of the present invention may take the form of acomputer program product embodied in one or more computer readablemedium(s) having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for aspects of thepresent invention may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

Aspects of the present invention are described below with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

Referring now to FIG. 1 and computer 10, in which an exemplary computersystem/server 12 which is applicable to implement the embodiments of thepresent invention is shown. Computer system/server 12 is onlyillustrative and is not intended to suggest any limitation as to thescope of use or functionality of embodiments of the invention describedherein.

As shown in FIG. 1, computer system/server 12 is shown in the form of ageneral-purpose computing device. The components of computersystem/server 12 may include, but are not limited to, one or moreprocessors or processing units 16, a system memory 28, and a bus 18 thatcouples various system components including system memory 28 toprocessor 16.

Bus 18 represents one or more of any of several types of bus structures,including a memory bus or memory controller, a peripheral bus, anaccelerated graphics port, and a processor or local bus using any of avariety of bus architectures. By way of example, and not limitation,such architectures include Industry Standard Architecture (ISA) bus,Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, VideoElectronics Standards Association (VESA) local bus, and PeripheralComponent Interconnect (PCI) bus.

Computer system/server 12 typically includes a variety of computersystem readable media. Such media may be any available media that isaccessible by computer system/server 12, and it includes both volatileand non-volatile media, removable and non-removable media.

System memory 28 can include computer system readable media in the formof volatile memory, such as random access memory (RAM) 30 and/or cachememory 32. Computer system/server 12 may further include otherremovable/non-removable, volatile/non-volatile computer system storagemedia. By way of example only, storage system 34 can be provided forreading from and writing to a non-removable, non-volatile magnetic media(not shown and typically called a “hard drive”). Although not shown, amagnetic disk drive for reading from and writing to a removable,non-volatile magnetic disk (e.g., a “floppy disk”), and an optical diskdrive for reading from or writing to a removable, non-volatile opticaldisk such as a CD-ROM, DVD-ROM or other optical media can be provided.In such instances, each can be connected to bus 18 by one or more datamedia interfaces. As will be further depicted and described below,system memory 28 may include at least one program product having a set(e.g., at least one) of program modules that are configured to carry outthe functions of embodiments of the invention.

Program/utility 40, having a set (at least one) of program modules 42,may be stored in system memory 28 by way of example, and not limitation,as well as an operating system, one or more application programs, otherprogram modules, and program data. Each of the operating system, one ormore application programs, other program modules, and program data orsome combination thereof, may include an implementation of a networkingenvironment. Program modules 42 generally carry out the functions and/ormethodologies of embodiments of the invention as described herein.

Computer system/server 12 may also communicate with one or more externaldevices 14 such as a keyboard, a pointing device, a display 24, etc.;one or more devices that enable a user to interact with computersystem/server 12; and/or any devices (e.g., network card, modem, etc.)that enable computer system/server 12 to communicate with one or moreother computing devices. Such communication can occur via Input/Output(I/O) interfaces 22. Still yet, computer system/server 12 cancommunicate with one or more networks such as a local area network(LAN), a general wide area network (WAN), and/or a public network (e.g.,the Internet) via network adapter 20. As depicted, network adapter 20communicates with the other components of computer system/server 12 viabus 18. It should be understood that although not shown, other hardwareand/or software components could be used in conjunction with computersystem/server 12. Examples, include, but are not limited to: microcode,device drivers, redundant processing units, external disk drive arrays,RAID systems, tape drives, and data archival storage systems, etc.

FIG. 2 shows a flowchart of a method for evaluating a predictive modelaccording to an embodiment of the present invention. The predictivemodel to be evaluated may be created by using any data mining platformthat is currently available on markets. When a predictive model iscreated, a set of samples may be divided into two portions: a set ofsamples with known prediction results and a set of samples with unknownprediction results. In the field of data mining, typically, the set ofsamples with known prediction results is referred to as a set oftraining samples, while the set of samples with unknown predictionresults is referred to as a set of test samples. Then, a data miningalgorithm is applied to the set of training samples for creating apredictive model.

Hereinafter, there is provided a brief description of the process ofcreating a predictive model with reference to a specific example. Inthis example, a telecommunications service provider is concerned aboutthe amount of customers that will churn to competitors. If customers whopossibly churn to other providers can be predicted by using serviceusage data, they may be retained as much as possible by customizingservice usage data.

The telecommunications service provider has material and historicalusage information for 100,000 customers, and then a set of samples maybe built by using the material and historical usage information forthese customers, as shown by Table 1 in FIG. 7.

The set of samples may be randomly divided into two portions: a set oftraining samples (containing material and historical usage informationfor 80,000 customers) and a set of test samples (containing material andhistorical usage information for 20,000 customers). Then, on a datamining platform, a logistic regression training method is applied to theset of training samples so as to create a logistic regression model. Thelogistic regression model may be defined as below: given several markedsamples, each of which has attribute information [x1, x2, . . . , xn]and corresponding mark information y, the logistic regression modellearns a function y=f([x1, x2, . . . , xn]) so that fitting error ofthese samples is minimized. Here, y is discrete-valued, and x1, x2, . .. , xn may be either discrete values or continuous values. In performinga customer churn analysis, each customer may be regarded as a sample.Regarding previous customers, it can be known who are renewed and whoare not renewed (churned). Renewal or not is represented by y in thelogistic regression with a discrete value of “churned” or “non-churned”.Each customer has information such as area, vocation, age, monthlyincome, average monthly consumption, sex, WIFI business, average monthlylong-distance fee, service time, etc., just as shown by Table 1 in FIG.7. The information is represented by [x1, x2, x3 . . . , xn]. That is,each x corresponds to an attribute. Information for these customers isused as x, and a result indicating renewal or not is used as y; then,these <x,y> pairs are used as inputs to the predictive model, whereby acorresponding function f is trained such that total error of f(x) and yfor all samples is minimized.

It should be understood that although a predictive model has beendescribed in detail by taking as an example a logistic regression modelused for customer churn analyses in the telecommunications sector, basedon different applications of predictive models, the method forevaluating a predictive model according to the embodiments of thepresent invention is not limited thereto, but may be applied to otherindustries than telecommunications, and also may be aimed at othertechnical objectives than predicting customer churn. In addition, theabove-described process of creating a predictive model is exemplaryonly, and the implementation of the method of the present invention justrelies on an existing predictive model but is not limited to anyspecific process of creating a predictive model.

After creating the predictive model, the created predictive model may beevaluated by using the method according to the embodiments of thepresent invention.

As shown in FIG. 2, first in step S201, features of training samples ina set of training samples are identified. The features of the trainingsamples may comprise, for example, features indicating the type of adata mining task to which a training sample is directed, and featuresindicating the type of the training samples, etc. In one embodiment, thetype of the data mining task to which the training samples are directedis first identified, and then the type of the training samples for whichthe type of the directed data mining task has been identified isidentified. Types of the training samples comprise static, time series,etc. For example, a timestamp in the set of training samples mayindicate training samples are of a type of time series, while theabsence of a timestamp in the set of training samples may indicatetraining samples are static samples. In addition, types of data miningtasks usually comprise classification tasks (including bi-classificationtasks and multi-classification tasks), clustering tasks, estimationtasks, etc. The type of the data mining task to which training samplesare directed may be indicated by the presence or absence of a label. Forexample, if the set of training samples contains a label, it indicatesthe type of the data mining task to which training samples are directedis a classification task.

In step S202, at least one evaluation metric may be selected from a setof evaluation metrics as one or more available metrics based on theidentified features of the training samples. In one embodiment of thepresent invention, the set of evaluation metrics comprises a pluralityof first elements, each of which comprises an evaluation metric and atleast one attribute associated with the evaluation metric. The at leastone attribute associated with the evaluation metric comprises, forexample: a type of sample to which the evaluation metric is applicable,a type of a data mining task to which the evaluation metric isapplicable, a requirement of the evaluation metric on the ratio betweeninter-classification samples, etc. Table 2 in FIG. 8 shows a specificexample of the set of evaluation metrics.

It should be understood that Table 2 only lists a limited number ofevaluation metrics but does not exhaust all evaluation metrics used forevaluating a predictive model. The set of evaluation metrics accordingto the embodiments of the present invention includes, but is not limitedto the evaluation metrics listed in Table 2. In addition, Table 2 liststhree attributes, i.e. a type of samples to which evaluation metrics areapplicable, a type of the data mining task to which the evaluationmetrics are applicable, and requirements of the evaluation metrics onthe ratio between inter-classification samples. However, the set of theevaluation metrics according to the embodiments of the present inventionis not limited thereto, but may contain more or less attributes. Forexample, the attribute such as requirement on the ratio betweeninter-classification samples (the 4^(th) column) is optional.

In one embodiment, selecting at least one evaluation metric from a setof evaluation metrics as one or more available metrics comprises:comparing the identified features of the training samples with the atleast one attribute of each evaluation metric in the set of evaluationmetrics; and in response to the identified features of the trainingsamples matching at least one attribute of at least one evaluationmetric, selecting the at least one evaluation metric as the one or moreavailable metric.

In one embodiment, where only the type of the data mining task to whichthe training samples are directed is identified, the identified type ofthe data mining task to which the training samples are directed iscompared with the at least one attribute of each evaluation metric inthe set of evaluation metrics; in response to the identified type of thedata mining task to which the training samples are directed matching thetype of the data mining task to which one or more evaluation metrics inthe set of evaluation metrics are applicable, the one or more evaluationmetrics are selected as the one or more available metrics.

In one embodiment, where the type of the data mining task to which thetraining samples are directed is first identified and the type of thetraining samples whose type of the data mining task has been identifiedis identified subsequently, both the identified type of the data miningtask and the type of the training samples are compared with the at leastone attribute of each evaluation metric in the set of evaluationmetrics. In response to the type of the data mining task and the type ofthe training samples matching the type of the data mining task to whichone or more evaluation metrics in the set of evaluation metrics areapplicable and the type of samples to which the evaluation metrics areapplicable, the one or more evaluation metrics are selected as the oneor more available metric.

With continued reference to FIG. 2, in step S203, a predictive modelcreated based on the set of training samples is applied to a set of testsamples so as to calculate values of the available metrics. Hereinafter,with respect to the example of predicting telecommunications customerchurn as shown in Table 1, by taking the calculation of a value ofF-score, brief description is provided as to how to calculate values ofthe available metrics.

F-score may be given as below:F-score=(2*precision*recall)/(precision+recall)  (Equation 1)Precision=TP/(TP+FP)  (Equation 2)Recall=TP/(TP+FN)  (Equation 3)

where TP (True Positive) represents customers who will stop to usecompetitors' products and who are predicted precisely;

FP (False Positive) represents customers who will stop to usecompetitors' products and who are not predicted precisely;

FN (False Negative) represents customers who will continue using thiscompany's products and who are not predicted precisely; and

TN (True Negative) represents customers who will continue using thiscompany's products and who are predicted precisely.

A mapping table (Table 3) in FIG. 9 shows a prediction result obtainedby applying the predictive model to the set of test samples. By makingstatistics on the prediction result, it is calculated from Equations 2and 3 that Precision=0.75, Recall=0.79, and then it is calculated fromEquation 3 that F-score=0.77.

In step S204, the predictive model is evaluated by using the availablemetrics and the values of the available metrics. As is well known tothose skilled in the art, the process of evaluating the predictive modelby using the available metrics and the values of the available metricsmay be implemented on an existing data mining platform and thus thedetailed description of the process is omitted.

Hereinafter, with reference to Table 1, Table 2 and FIG. 3, there ispresented description of a flowchart of analyzing training samples andselecting one or more available metrics from a set of evaluation metricsbased on the analysis according to an embodiment of the presentinvention. In step S301, a type of a data mining task to which trainingsamples in a set of training samples are directed is identified. Forexample, if it is identified that the set of training samples contains alabel (e.g. as shown by Table 1), it is indicated the training samplesare directed to a classification task (as shown by block A). If it isidentified that the set of training samples contains no label, it isusually considered that the training samples are directed to aclustering task (as shown by block B). In step S302, the number of theclassifications is identified. If a value of the label is 0 or 1, thenumber of the classifications may be identified as 2, i.e., the trainingsample is directed to a bi-classification task. For example, as shown bythe last column in Table 1, the prediction result only comprises twoclassifications, i.e., churned and non-churned, so it may be determinedthat the training samples are directed to a bi-classification task. Instep S303, the ratio between inter-classification samples is calculated.For example, in the example as shown by Table 1, it is calculated thatthe ratio of lost customers to non-lost customers is 1:9. Then, in stepS304, one or more evaluation metrics, such as AUC, Accuracy, F-score,Recall and Precision, that are applicable to the bi-classification taskare selected from the set of evaluation metrics as shown by Table 2 asone or more available metrics. While the number of classifications isidentified in step S302, if the label has more than two values, thenumber of classifications is identified as more than 2, i.e., thetraining samples are directed to a multi-classification task. As aresult, in step S304, one or more evaluation metrics, such as Accuracyand MAUC, which are applicable to the multi-classification task areselected from the set of evaluation metrics as shown in Table 2 as oneor more available metrics.

While a type of a data mining task to which training samples in a set oftraining samples are directed is identified in step S301, if it isidentified that the set of training samples contains no label, it isusually considered that the training samples are directed to aclustering task (as shown by block B). Then in step S305, the ratiobetween inter-classification samples is calculated. In step S306, a typeof the training samples is identified. If the type of the trainingsample is identified as time series type, one or more evaluationmetrics, such as Sum of Squares Error, that are applicable to theclustering task and to the training samples of the time series type areselected from the set of evaluation metrics as shown in Table 2 as oneor more available metrics in step S304.

FIG. 4 shows a flowchart of a method for evaluating a predictive modelaccording to another embodiment of the present invention. Besides stepsS401, S402 and S403 that are identical to steps S201, S202 and S203 inFIG. 2, the method as shown in FIG. 4 further comprises steps S404, S405and S406. In step S404, one or more recommended metrics are determinedbased on the predictive model, the available metrics and a predeterminedfirst set of user-preferred metrics. In one embodiment, the first set ofuser-preferred metrics comprises a plurality of second elements, each ofwhich comprises a first user-preferred metric and at least one attributeassociated with the first user-preferred metric. The firstuser-preferred metrics contained in the first set of user-preferredmetrics may be identical to evaluation metrics contained in the set ofevaluation metrics or comprise a subset thereof. The at least oneattribute associated with the first user-preferred metric at leastcomprises an attribute indicating a degree of a user's preference to thefirst user-preferred metric, e.g. weight. Table 4 in FIG. 10 shows aspecific example of the first set of user-preferred metrics.

The first set of user-preferred metrics as shown in Table 4 comprises aplurality of first user-preferred metrics, such as AUC, Accuracy,F-score, Recall, Precision, etc. Attributes associated with the firstuser-preferred metrics comprise an attribute indicating industries (e.g.communications, banking) to which the first user-preferred metrics areapplied, an attribute indicating application scenarios (e.g. customeranalysis, base station failure prediction in communications; credit riskprediction in banking) of the first user-preferred metrics in a specificindustry, and a weight indicating the degree of the user's preference toa specific first user-preferred metric. The larger the weight is, thegreater the degree of the user's preference to the metric would be. Forexample, in customer analysis scenarios in the communications industry,a weight of AUC is 0.9, while a weight of Accuracy is 0.1, whichindicates that the user prefers to use AUC for evaluating the predictivemodel.

In the embodiments described above with reference to Table 1, Table 2and FIG. 3, in order to evaluate the predictive model created based onthe set of training samples in Table 1, five evaluation metrics areselected from the set of evaluation metrics in Table 2 as availablemetrics, i.e., AUC, Accuracy, F-score, Recall, and Precision. In anembodiment of the present invention, these available metrics may befurther compared with first user-preferred metrics in the first set ofuser-preferred metrics in Table 4, and first user-preferred metricsmatching these available metrics are selected from Table 4 asrecommended metrics to be recommended to the user. Specifically, anindustry (communications) to which the training samples as shown inTable 1 are applied is determined based on, for example, interactionwith the user. Then, an application scenario (customer analysis) in theindustry of the training samples as shown in Table 1 is determined basedon the set of training samples, the analysis of the created predictivemodel and the interaction with the user. Subsequently, the availablemetrics AUC, Accuracy, F-score, Recall and Precision and the determinedindustry and application scenario to which the training samples areapplied are compared with the first user-preferred metrics and theassociated attributes (application industry, application scenario) asshown in Table 4, so as to select from Table 4 the first user-preferredmetrics AUC, Accuracy, F-score, Recall and Precision matching theavailable metrics AUC, Accuracy, F-score, Recall and Precision as therecommended metrics.

In one embodiment, selecting one or more first user-preferred metricsmatching the one or more available metrics from the first set ofuser-preferred metrics as the one or more recommended metrics comprisesranking the selected first user-preferred metrics by weight in responseto a user input indicating a desired number of recommended metrics; andsequentially selecting the number of first user-preferred metrics fromthe ranked first user-preferred metrics as the one or more recommendedmetrics. For example, if the user inputs 4 as a desired number of therecommended metrics, in response to the user input, the firstuser-preferred metrics AUC, Accuracy, F-score, Recall and Precisionselected from Table 4 are ranked in decreasing order by weight, and thenthe first 4 first user-preferred metrics AUC, F-score, Recall andPrecision are used as the recommended metrics. Further, weightsassociated with the selected first user-preferred metrics may bedetermined as weights of the recommended metrics, and the recommendedmetrics are presented to the user in association with the weights of therecommended metrics.

Returning to FIG. 4, in step S405, values of the recommended metrics aredetermined based on the values of the available metrics. In other words,the values of the available metrics corresponding to the recommendedmetrics are determined as the values of the recommended metrics. In stepS406, the predictive model is evaluated by using the recommended metricsand the values of the recommended metrics.

FIG. 5 shows a flowchart of a method for evaluating a predictive modelaccording to a further embodiment of the present invention. Besidessteps S501, S502 and S503 that are identical to steps S201, S202 andS203 in FIG. 2, and steps S504 and S505 that are identical to steps S404and S405 in FIG. 4, the method as shown in FIG. 5 further comprisessteps S506 to S510. In step S506, a second set of user-preferred metricsis received from the user. In one embodiment, the second set ofuser-preferred metrics comprises a plurality of third elements, each ofwhich at least comprises a second user-preferred metric and a weightassociated with the second user-preferred metric. Second user-preferredmetrics contained in the second set of user-preferred metrics may be asubset of the first user-preferred metrics contained in the first set ofuser-preferred metrics. Table 5 in FIG. 11 shows a specific example ofthe second set of user-preferred metrics.

In step S507, the recommended metrics are compared with seconduser-preferred metrics in the second set of user-preferred metrics. Instep S508, one or more second user-preferred metrics matching therecommended metrics are selected from the second set of user-preferredmetrics. In step S509, the values of the recommended metrics matchingthe selected second user-preferred metrics are determined as values ofthe selected second user-preferred metrics. In step S510, the predictivemodel is evaluated by using the selected user-preferred metrics and thevalues of the second user-preferred metrics. With continued reference tothe example as described above, the recommended metrics AUC, F-score,Recall and Precision selected from Table 4 are compared with the seconduser-preferred metrics in Table 5, and then the second user-preferredmetrics matching the recommended metrics AUC, F-score, Recall areselected from Table 5.

In one embodiment, the selected second user-preferred metrics are rankedby weight, and the ranked second user-preferred metrics are presented tothe user in association with the weights of the second user-preferredmetrics.

In one embodiment, the first set of user-preferred metrics is updated byusing the second user-preferred metrics and weights of the seconduser-preferred metrics. The updating may be a heuristic progressiveupdating. For example, in the second set of user-preferred metrics asshown in Table 5, the weight 0.9 of F-score is larger than the weight0.3 of AUC, which indicates that in customer analysis scenarios in thecommunications industry, users prefer to use F-score for evaluatingpredictive models. After receiving Table 5 from the user, weights of themetrics F-score, Recall and AUC in Table 4 may be modified according toa predetermined step size based on the metrics F-score, Recall and AUCas well as their associated weights in Table 5. For example, the weightof F-score in Table 4 is increased by 0.1, while the weight of AUC isdecreased by 0.1, whereby Table 4 is updated. The updated Table 4 may beused subsequently.

In a second aspect, embodiments of the present invention provide anapparatus for evaluating a predictive model. FIG. 6 shows a blockdiagram of an apparatus for evaluating a predictive model according toan embodiment of the present invention. As shown in FIG. 6, theapparatus for evaluating a predictive model comprises: an identifyingmodule 601 configured to identify features of the training samples in aset of training samples; a selecting module 602 configured to select atleast one evaluation metric from a set of evaluation metrics as one ormore available metrics based on the identified features; a calculatingmodule 603 configured to apply a predictive model created based on theset of training samples to a set of test samples so as to calculatevalues of the available metrics; and an evaluating module 604 configuredto evaluate the predictive model by using the available metrics and thevalues of the available metrics.

In an embodiment, the set of evaluation metrics comprises a plurality offirst elements, each of which comprises an evaluation metric and atleast one attribute associated with the evaluation metric.

In an embodiment, the selecting module 602 comprises: a first comparingunit configured to compare the identified features with the at least oneattribute of each evaluation metric in the set of evaluation metrics;and a first selecting unit configured to, in response to the identifiedfeatures matching at least one attribute of at least one evaluationmetric, select the at least one evaluation metric as the availablemetrics.

In an embodiment, the at least one attribute associated with theevaluation metric at least comprises a type of samples to which theevaluation metric is applicable, and a type of a data mining task towhich the evaluation metric is applicable.

In an embodiment, the identified features at least comprise a type ofthe training samples and a type of a data mining task to which thetraining samples are directed.

In an embodiment, the selecting module 602 comprises a first determiningmodule configured to determine one or more recommended metrics based onthe predictive model, the available metrics and a predetermined firstset of user-preferred metrics; the calculating module 603 comprises asecond determining module configured to determine values of therecommended metrics based on the values of the available metric; and theevaluating module 604 is further configured to evaluate the predictivemodel by using the recommended metrics and the values of the recommendedmetrics.

In an embodiment, the first set of user-preferred metrics comprises aplurality of second elements, each of which comprises a firstuser-preferred metric and at least one attribute associated with thefirst user-preferred metric, and wherein the at least one attributeassociated with the first user-preferred metric at least comprises aweight indicating a degree of a user's preference to the firstuser-preferred metric.

In an embodiment, the first determining module comprises: a secondcomparing unit configured to compare the available metrics with firstuser-preferred metrics in the first set of user-preferred metrics; asecond selecting unit configured to select one or more firstuser-preferred metrics matching the available metrics from the first setof user-preferred metrics; a first ranking unit configured to, inresponse to a user's input indicating a desired number of therecommended metrics, rank the selected first user-preferred metrics byweight; and a sequentially selecting unit configured to sequentiallyselect the desired number of the first user-preferred metrics as the oneor more recommended metrics from the ranked first user-preferredmetrics.

In an embodiment, the first determining module further comprises: areceiving module configured to receive a second set of user-preferredmetrics from a user; a comparing module configured to compare therecommended metrics with second user-preferred metrics in the second setof user-preferred metrics; and a user-preferred metric selecting moduleconfigured to select one or more second user-preferred metrics matchingthe recommended metrics from the second set of user-preferred metrics.In this embodiment, the second determining module is further configuredto determine the values of the recommended metrics matching the selectedsecond user-preferred metrics as values of the selected seconduser-preferred metrics, and the evaluating module is further configuredto evaluate the predictive model by using the selected seconduser-preferred metrics and the values of the selected seconduser-preferred metrics.

In an embodiment, the second set of user-preferred metrics comprises aplurality of third elements, each of which at least comprises a seconduser-preferred metric and a weight associated with the seconduser-preferred metric.

In an embodiment, the apparatus further comprising: an updating moduleconfigured to update the first set of user-preferred metrics by usingthe second user-preferred metrics and the weights of the seconduser-preferred metrics.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

The descriptions of the various embodiments of the present inventionhave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

The invention claimed is:
 1. A method for evaluating a predictive model,comprising: identifying, by one or more computer processors, features oftraining samples in a set of training samples; selecting, by one or morecomputer processors, at least one evaluation metric from a set ofevaluation metrics as one or more available metrics based on theidentified features, the selecting including determining one or morerecommended metrics based on the predictive model, the one or moreavailable metrics, and a predetermined first set of user-preferredmetrics, wherein the predetermined first set of user-preferred metricsincludes a plurality of second elements, each of which comprises a firstuser-preferred metric and at least one attribute associated with thefirst user-preferred metric, and wherein the at least one attributeassociated with the first user-preferred metric at least comprises aweight indicating a degree of a user's preference to the firstuser-preferred metric; applying, by one or more computer processors, thepredictive model created based on the set of training samples to a setof test samples so as to calculate values of the one or more availablemetrics, wherein applying, by one or more computer processors, thepredictive model created based on the set of training samples to a setof test samples so as to calculate values of the one or more availablemetrics comprises determining values of the one or more recommendedmetrics based on the values of the one or more available metrics; andevaluating, by one or more computer processors, the predictive model byusing the one or more available metrics and the values of the one ormore available metrics, wherein evaluating, by one or more computerprocessors, the predictive model comprises evaluating the predictivemodel by using the one or more recommended metrics and the values of theone or more recommended metrics.
 2. The method according to claim 1,wherein the set of evaluation metrics comprises a plurality of firstelements, each of which comprises an evaluation metric and at least oneattribute associated with the evaluation metric.
 3. The method accordingto claim 2, wherein selecting, by one or more computer processors, atleast one evaluation metric from the set of evaluation metrics as one ormore available metrics based on the identified features comprises:comparing, by one or more computer processors, the identified featureswith the at least one attribute associated with each evaluation metricin the set of evaluation metrics; and in response to the identifiedfeatures matching at least one attribute of at least one evaluationmetric in the set of evaluation metrics, selecting, by one or morecomputer processors, the at least one evaluation metric as the one ormore available metrics.
 4. The method according to claim 2, wherein theat least one attribute associated with the evaluation metric at leastcomprises a type of samples to which the evaluation metric isapplicable, and a type of a data mining task to which the evaluationmetric is applicable.
 5. The method according to claim 1, wherein theidentified features at least comprise a type of the training samples anda type of a data mining task to which the training samples are directed.6. The method according to claim 1, wherein determining, by one or morecomputer processors, one or more recommended metrics comprises:comparing, by one or more computer processors, the one or more availablemetrics with first user-preferred metrics in the predetermined first setof user-preferred metrics; selecting, by one or more computerprocessors, one or more first user-preferred metrics matching the one ormore available metrics from the first set of user-preferred metrics; inresponse to a user's input indicating a desired number of the one ormore recommended metrics, ranking, by one or more computer processors,the selected one or more first user-preferred metrics by weight; andsequentially selecting, by one or more computer processors, the desirednumber of the one or more first user-preferred metrics as the one ormore recommended metrics from the ranked one or more firstuser-preferred metrics.
 7. The method according to claim 1, whereindetermining one or more recommended metrics further comprises:receiving, by one or more computer processors, a second set ofuser-preferred metrics from a user; comparing, by one or more computerprocessors, the one or more recommended metrics with seconduser-preferred metrics in the second set of user-preferred metrics; andselecting, by one or more computer processors, one or more seconduser-preferred metrics matching the one or more recommended metrics fromthe second set of user-preferred metrics; wherein determining values ofthe one or more recommended metrics comprises: determining, by one ormore computer processors, the values of the one or more recommendedmetrics matching the selected second user-preferred metrics as values ofthe selected second user-preferred metrics, and wherein evaluating, byone or more computer processors, the predictive model comprises:evaluating, by one or more computer processors, the predictive model byusing the selected second user-preferred metrics and the values of theselected second user-preferred metrics.
 8. The method according to claim7, wherein the second set of user-preferred metrics comprises aplurality of third elements, each of which at least comprises a seconduser-preferred metric and a weight associated with the seconduser-preferred metric.
 9. The method according to claim 8, furthercomprising: updating, by one or more computer processors, the first setof user-preferred metrics by using the second user-preferred metrics andthe weight of the second user-preferred metric.
 10. A computer programproduct for evaluating a predictive model, the computer program productcomprising: one or more computer readable storage media and programinstructions stored on the one or more computer readable storage media,the program instructions executable by a processor, the programinstructions comprising instructions for: identifying features oftraining samples in a set of training samples; selecting at least oneevaluation metric from a set of evaluation metrics as one or moreavailable metrics based on the identified features, the selectingincluding determining one or more recommended metrics based on thepredictive model, the one or more available metrics, and a predeterminedfirst set of user-preferred metrics, wherein the predetermined first setof user-preferred metrics comprises a plurality of second elements, eachof which comprises a first user-preferred metric and at least oneattribute associated with the first user-preferred metric, and whereinthe at least one attribute associated with the first user-preferredmetric at least comprises a weight indicating a degree of a user'spreference to the first user-preferred metric; applying the predictivemodel created based on the set of training samples to a set of testsamples so as to calculate values of the one or more available metrics,wherein applying the predictive model created based on the set oftraining samples to a set of test samples so as to calculate values ofthe one or more available metrics comprises determining values of theone or more recommended metrics based on the values of the one or moreavailable metrics; and evaluating the predictive model by using the oneor more available metrics and the values of the one or more availablemetrics, wherein evaluating the predictive model comprises evaluatingthe predictive model by using the one or more recommended metrics andthe values of the one or more recommended metrics.
 11. The computerprogram product of claim 10, wherein the set of evaluation metricscomprises a plurality of first elements, each of which comprises anevaluation metric and at least one attribute associated with theevaluation metric.
 12. The computer program product of claim 11, whereinselecting at least one evaluation metric from a set of evaluationmetrics as one or more available metrics based on the identifiedfeatures comprises: comparing the identified features with the at leastone attribute associated with each evaluation metric in the set ofevaluation metrics; and in response to the identified features matchingat least one attribute of at least one evaluation metric in the set ofevaluation metrics, selecting the at least one evaluation metric as theone or more available metrics.
 13. The computer program product of claim11, wherein the at least one attribute associated with the evaluationmetric at least comprises a type of samples to which the evaluationmetric is applicable, and a type of a data mining task to which theevaluation metric is applicable.
 14. The computer program product ofclaim 10, wherein the identified features at least comprise a type ofthe training samples and a type of a data mining task to which thetraining samples are directed.
 15. A computer system for evaluating apredictive model, the computer system comprising: one or more computerprocessors; one or more computer readable storage media; and programinstructions stored on the one or more computer readable storage mediafor execution by at least one of the one or more computer processors,the program instructions comprising instructions for: identifyingfeatures of training samples in a set of training samples; selecting atleast one evaluation metric from a set of evaluation metrics as one ormore available metrics based on the identified features, the selectingincluding determining one or more recommended metrics based on thepredictive model, the one or more available metrics, and a predeterminedfirst set of user-preferred metrics, wherein the predetermined first setof user-preferred metrics comprises a plurality of second elements, eachof which comprises a first user-preferred metric and at least oneattribute associated with the first user-preferred metric, and whereinthe at least one attribute associated with the first user-preferredmetric at least comprises a weight indicating a degree of a user'spreference to the first user-preferred metric; applying the predictivemodel created based on the set of training samples to a set of testsamples so as to calculate values of the one or more available metrics,wherein applying the predictive model created based on the set oftraining samples to a set of test samples so as to calculate values ofthe one or more available metrics comprises determining values of theone or more recommended metrics based on the values of the one or moreavailable metrics; evaluating the predictive model by using the one ormore available metrics and the values of the one or more availablemetrics, wherein evaluating the predictive model comprises evaluatingthe predictive model by using the one or more recommended metrics andthe values of the one or more recommended metrics.
 16. The computersystem of claim 15, wherein the set of evaluation metrics comprises aplurality of first elements, each of which comprises an evaluationmetric and at least one attribute associated with the evaluation metric.17. The computer system of claim 15, wherein determining one or morerecommended metrics further comprises: receiving a second set ofuser-preferred metrics from a user; comparing the one or morerecommended metrics with second user-preferred metrics in the second setof user-preferred metrics; and selecting one or more seconduser-preferred metrics matching the one or more recommended metrics fromthe second set of user-preferred metrics; wherein determining values ofthe one or more recommended metrics comprises: determining the values ofthe one or more recommended metrics matching the selected seconduser-preferred metrics as values of the selected second user-preferredmetrics, and wherein evaluating the predictive model comprises:evaluating the predictive model by using the selected seconduser-preferred metrics and the values of the selected seconduser-preferred metrics.
 18. The computer system of claim 17, wherein thesecond set of user-preferred metrics comprises a plurality of thirdelements, each of which at least comprises a second user-preferredmetric and a weight associated with the second user-preferred metric.